Papers with Expected Calibration Error
SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a profound impact on a wide range of applications. |
| Approach: | They propose a framework to mitigate the tool-abuse behavior of Large Language Models and propose SMARTCAL to mitigate this issue. |
| Outcome: | The proposed framework improves the performance of LLMs on three datasets with two mainstream tool-use frameworks and shows an 8.6% increase in QA performance and 21.6 percent lower expected calibration error (ECE) than existing methods. |
SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models have a tendency to make confidently wrong predictions, highlighting the need for uncertainty quantification (UQ) . previous studies focused on aleatoric uncertainty, but the full spectrum of uncertainties, including epistemic, remains inadequately explored. |
| Approach: | They propose a method to quantify uncertainty in large language models (LLMs) they use a set of perturbations and an aggregation module to generalize the method. |
| Outcome: | The proposed method improves model uncertainty calibration and reduces expected calibration error by 50% on average. |
BaseCal: Unsupervised Confidence Calibration via Base Model Signals (2026.acl-long)
Copied to clipboard
Hexiang Tan, Wanli Yang, Junwei Zhang, Xin Chen, Rui Tang, Du Su, Jingang Wang, Yuanzhuo Wang, Fei Sun, Xueqi Cheng
| Challenge: | Post-trained LLMs typically compromise reliability with severe overconfidence, resulting in inaccurate responses. |
| Approach: | They propose a solution that feeds PoLLMs into the base LLM to get confidence. |
| Outcome: | The proposed solution reduces expected calibration error (ECE) by 42.90% compared to the best unsupervised baselines. |
NeuroSym-Cal: Bridging the Reasoning-Execution Gap in Code Generation via Hierarchical Calibration (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing calibration methods rely on the assumption that consensus implies correctness . Existing methods fail under systematic errors, leading to miscalibrated high-confidence predictions. |
| Approach: | They propose a hierarchical calibration framework that measures confidence at two levels . they propose sensitivity analysis to measure local curvature of deductive process . |
| Outcome: | The proposed framework de-saturates overconfident errors and improves selective generation performance on OOD benchmarks. |
Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on the confidence calibration of LLMs have not explored the effects of different prompting strategies on LLM performance. |
| Approach: | They propose Fact-and-Reflection prompting which improves LLM confidence calibration . they propose to use human cognition to elicit known "facts" and ask model to "reflect" over them . |
| Outcome: | The proposed method lowers the expected calibration error by 23.5% on multi-purpose QA tasks. |
Calibrating LLM Confidence by Probing Perturbed Representation Stability (2025.emnlp-main)
Copied to clipboard
Reza Khanmohammadi, Erfan Miahi, Mehrsa Mardikoraem, Simerjot Kaur, Ivan Brugere, Charese Smiley, Kundan S Thind, Mohammad M. Ghassemi
| Challenge: | Despite their impressive performance, large language models (LLMs) consistently struggle with confidence calibration. |
| Approach: | They propose a method to analyze internal representational stability in large language models by applying adversarial perturbations to final hidden states and using a lightweight classifier to predict answer correctness. |
| Outcome: | CCPS significantly outperforms existing methods on LLMs from 8B to 32B parameters in multiple-choice and open-ended formats. |
On Diversified Preferences of Large Language Model Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be fine tuned with human feedback, but human preferences can be diversified due to annotators’ different tastes, which hinders the effectiveness of LLM alignment methods. |
| Approach: | They propose a calibration error metric to evaluate large language models (LLMs) and a multi-objective reward learning method to enhance the calibration performance of RMs on shared preferences. |
| Outcome: | The proposed model can be adopted as a key calibration error and MORE can achieve superior alignment performance. |
DisCal: Distribution-Aware Calibration for Mathematical Reasoning Under Character-Level Noisy Inputs (2026.acl-long)
Copied to clipboard
Bo Zhang, Jiawei Zhang, Cong Gao, Bingxu Han, Minghao Hu, Jun Zhang, Yunbo Cao, Zhunchen Luo, Wen Yao, Guotong Geng, Zhong Wang
| Challenge: | Existing methods for calibration of large reasoning models (LRMs) focus on clean inputs, leaving noise unexplored. |
| Approach: | They propose a confidence calibration framework for character-level noisy inputs that extracts uncertainty signals from both the empirical answer distribution and the model’s predictive distribution and integrates them via a learned calibrator. |
| Outcome: | Experiments on multiple mathematical reasoning benchmarks show that DisCal outperforms existing calibration methods under noisy inputs, reducing expected calibration error (ECE) by up to 39.21% and improving Area Under the Receiver Operating Characteristic Curve (AUROC) by 31.44%. |
ODASim: Ordered, Distinctive and Absolute Semantic Similarity for Code Explanation Evaluation (2026.findings-acl)
Copied to clipboard
Prince Kumar, Vitobha Munigala, Jaydeep Sen, Ashish Mittal, Vishwajeet Kumar, Srikanth G. Tamilselvam
| Challenge: | Existing methods for code explanations fail to distinguish correct from partially or fully incorrect explanations and their similarity scores are poorly calibrated. |
| Approach: | They propose a model-agnostic graded fine-tuning framework that learns calibrated similarity representations between code and explanations to support fine-grained supervision and evaluation. |
| Outcome: | The proposed framework improves F1 score and ECE scores on two embedding models and reduces expected calibration error. |
Confidence Should Be Calibrated More Than One Turn Deep (2026.acl-long)
Copied to clipboard
| Challenge: | Existing work on confidence estimation and calibration focuses on single-turn settings . existing work on multi-turn calibration ignores the risks and potential of multi-turned conversations . |
| Approach: | They propose a multi-turn calibration task that reframes calibration from a static property into a dynamic challenge central to reliable multi- turn conversations. |
| Outcome: | The proposed model minimizes ECE@T and leverages ConfChat to improve confidence . the proposed model preserves and even enhances model performance in multi-turn interactions. |
When High Accuracy Hides Poor Calibration: Rethinking Confidence Evaluation in Transformer-Based Text Classification with Balanced Brier Score (2026.acl-long)
Copied to clipboard
Guilherme Fonseca, Gabriel Prenassi, Washington Cunha, Leonardo Chaves Dutra da Rocha, Marcos André Gonçalves
| Challenge: | Existing evidence for TC under fine-tuning is limited. |
| Approach: | They propose a calibration method that balances the contribution of correct and incorrect predictions within confidence bins. |
| Outcome: | The proposed calibration measures show that the models are overconfident even when miscalibrated . the proposed calibration methods challenge calibration assessment practices and provide a more reliable alternative for evaluating confidence quality in Transformer-based TC. |